Skip to content

feat: add server conformance tests for SEP-2575#271

Open
Yuan325 wants to merge 1 commit into
modelcontextprotocol:mainfrom
Yuan325:sep-2575
Open

feat: add server conformance tests for SEP-2575#271
Yuan325 wants to merge 1 commit into
modelcontextprotocol:mainfrom
Yuan325:sep-2575

Conversation

@Yuan325
Copy link
Copy Markdown

@Yuan325 Yuan325 commented May 12, 2026

Add conformance tests for SEP-2575 to validate the stateless-MCP behavior.

Motivation and Context

Ensure that servers adhere to the SEP.

How Has This Been Tested?

Tested locally

Breaking Changes

n/a. This tests are for the draft specs.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

server conformance tests for #266

Copy link
Copy Markdown
Member

@pcarleton pcarleton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — covers a good chunk of the version-negotiation and discover surface. (reminder the full SEP-2575 traceability YAML in #273 as a shared reference for this and #270.)

Coverage: the YAML has 20 server-side check: rows; this PR covers 7 (discover, version-error, 400-on-unsupported, header-mismatch, 404-on-unknown, missing-capability ×2). The big uncovered areas are subscriptions/listen (4 rows), HTTP cancellation (3 rows - 2 HTTP, 1 stdio), the core statelessness invariants (no-prior-context / no-connection-reuse), and server-no-log-without-loglevel. Fine to land those as follow-ups, but worth flagging which are intentionally deferred.

Correctness:

  • MUST→WARNING severitystateless-error-missing-capability and stateless-http-missing-capability use WARNING on failure, but the spec text is MUST for both -32003 and HTTP 400. If the worry is the test tool not existing, skip the check rather than downgrade.
  • stateless-version-response-header is dead — only ever pushed in a catch block (stateless.ts:360), never in the happy path. Also no spec backing: there's no server-MUST-return-header. Suggest dropping it.
  • Registered twice in index.ts (both pendingClientScenariosList and allClientScenariosList). We should clean this up, but you can put it in all and we filter it by spec version tag.
  • "Absent header" test isn't absent{'MCP-Protocol-Version': ''} sets an empty-string header; the helper always injects the default, so the header is never truly omitted. Needs a delete-key path in sendRpc.
  • stateless-version-header-mismatch only checks HTTP 400; spec says it SHOULD include -32001 HeaderMismatch in the body.
  • stateless-removed-* (5 checks) — there's no normative MUST forbidding ping/initialize; the only relevant text is the generic "unknown method → 404 + -32601". These checks don't assert the 404, and the example returns 200 for them. Suggest folding into one sep-2575-http-server-method-not-found-404 slug parametrized by method.
  • stateless-meta-missing* (4 checks) — the draft spec pages only put the MUST on the client (the SEP doc has a server-MUST-reject, but it didn't make it into basic/index.mdx — possibly a spec bug to fix). Either way, one check slug with the field name in details is probably better here.
  • noDrift requires identical array order between discover and error; spec doesn't mandate order — set comparison.
  • Check IDs — should be sep-2575-* to match #273 (same note left on #270); specReferences should point to spec anchors not the SEP PR.

Structure: ~60% of the 749 lines is duplicated catch-block boilerplate re-pushing id/name/description. A runCheck(id, desc, fn) helper would cut it to ~250 lines and prevent the try/catch description drift (e.g. stateless-version-unsupported has different descriptions in try vs catch).

Negative test could also be tightened — expect(failures.length).toBeGreaterThan(0) doesn't pin which checks fail; #270 uses expectedFailureSlugs which is the pattern we want.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants